Sparsity vs. Large Margins for Linear Classifiers

We provide small sample size bounds on the generalisation error of linear classiiers that take advantage of large observed margins on the training set and sparsity in the data dependent expansion coeecients. It is already known from results in the luckiness framework that both criteria independently have a large impact on the generalisation error. Our new results show that they can be combined which theoretically justiies learning algorithms like the Support Vector Machine 4] or the Relevance Vector Machine 12]. In contrast to previous studies we avoid using the classical technique of symmetrisation by a ghost sample but directly using the sparsity for the estimation of the generalisation error. We demonstrate that our result leads to practical useful results even in case of small sample size if the training set witnesses our prior belief in sparsity and large margins.